[FLINK-21309][metrics] Add UUID as grouping key for prom pushgateway by qinghui-xu · Pull Request #27744 · apache/flink

qinghui-xu · 2026-03-06T15:39:45Z

What is the purpose of the change

Previously we are relying on random uuid suffix of job label to avoid metric collisions among taskmangers. This makes it difficult to aggregate over job label.
Now we use the uuid as grouping key (flink_prometheus_push_gateway_reporter_id) instead, while respecting prometheus guide line on job label usage.

Brief change log

Remove the randomJobNameSuffix flag from the config options
job label will not be suffixed
Systematically add flink_prometheus_push_gateway_reporter_id = UUID as last grouping key in all case.

Verifying this change

Please make sure both new and modified tests in this PR follow the conventions for tests defined in our code quality guide.

This change added tests and can be verified as follows:

Added unit test to make sure that flink_prometheus_push_gateway_reporter_id = UUID is appended to user provided grouping keys.

Does this pull request potentially affect one of the following parts:

Dependencies (does it add or upgrade a dependency): (no)
The public API, i.e., is any changed class annotated with @Public(Evolving): (no)
The serializers: (no)
The runtime per-record code paths (performance sensitive): (no)
Anything that affects deployment or recovery: JobManager (and its components), Checkpointing, Kubernetes/Yarn, ZooKeeper: (no)
The S3 file system connector: (no)

Documentation

Does this pull request introduce a new feature? (no)
If yes, how is the feature documented? (not applicable)

flinkbot · 2026-03-06T15:46:17Z

CI report:

6fbca7e Azure: FAILURE

Bot commands

The @flinkbot bot supports the following commands:

@flinkbot run azure re-run the last Azure build

Previously we are relying on random uuid suffix of `job` label to avoid metric collisions among taskmangers. This makes it difficult to aggregate over `job` label. Now we use the uuid as grouping key instead, while respecting prometheus guide line on `job` label usage.

qinghui-xu · 2026-03-07T21:45:50Z

@flinkbot run azure

rionmonster · 2026-03-08T22:31:46Z

...metheus/src/main/java/org/apache/flink/metrics/prometheus/PrometheusPushGatewayReporter.java

 */
 @PublicEvolving
 public class PrometheusPushGatewayReporter extends AbstractPrometheusReporter implements Scheduled {
+    public static final String REPORTER_ID_GROUPPING_KEY =


Minor spelling nit: I think we want this to be called `REPORTER_ID_GROUPING_KEY.

rionmonster · 2026-03-08T22:39:44Z

.../src/main/java/org/apache/flink/metrics/prometheus/PrometheusPushGatewayReporterOptions.java

                    .defaultValue("")
                    .withDescription("The job name under which metrics will be pushed");

-    public static final ConfigOption<Boolean> RANDOM_JOB_NAME_SUFFIX =


Are we sure we want to remove this configuration entirely? I’d imagine there are existing jobs and/or users already relying on it. This would also require updating the associated documentation and, at a minimum, feels like something that should be deprecated before being removed.

Good point. Let me deprecate it in the PR. I'm wondering should I also change the default value to false while keeping it. I'll use this flag as a switch: if job is not suffixed, a "flink_prometheus_push_gateway_reporter_id" grouping key will be used.

Sent a followup commit in this sense.

rionmonster · 2026-03-08T22:49:25Z

...metheus/src/main/java/org/apache/flink/metrics/prometheus/PrometheusPushGatewayReporter.java

+
+        GroupingKeyMap(Map<String, String> customGroupingKeys) {
+            if (customGroupingKeys.containsKey(REPORTER_ID_GROUPPING_KEY)) {
+                throw new IllegalArgumentException(


We'll want to add a test here to cover the assertion against the user supplying "flink_prometheus_push_gateway_reporter_id" as an explicit grouping (asserting the expected exception).

- Keep `randomJobNameSuffix` config as deprecated - When `randomJobNameSuffix` is disabled (by default), group metrics using random reporter id

featzhang · 2026-03-17T13:48:04Z

...metheus/src/main/java/org/apache/flink/metrics/prometheus/PrometheusPushGatewayReporter.java

+     * grouping keys, so that each taskmanger instance and jobmanager will not collide their metrics
+     * in the PushGateway.
+     */
+    static class GroupingKeyMap extends AbstractMap<String, String> {


The custom GroupingKeyMap feels heavier than necessary for what is effectively appending one synthetic entry.

Since the goal is just to preserve user-defined grouping keys and add REPORTER_ID_GROUPING_KEY as the last entry, would a LinkedHashMap copy be simpler and easier to reason about?

For example:

Map<String, String> grouping = new LinkedHashMap<>(groupingKey);
grouping.put(REPORTER_ID_GROUPING_KEY, reporterId);

That avoids reimplementing AbstractMap, iterator behavior, and custom entrySet() semantics.

featzhang · 2026-03-17T13:49:00Z

...metheus/src/main/java/org/apache/flink/metrics/prometheus/PrometheusPushGatewayReporter.java

+                        "Grouping keys must not contain the reserved key: "
+                                + REPORTER_ID_GROUPPING_KEY);
+            }
+            this.customGroupingKeys = customGroupingKeys;


Would it make sense to defensively copy customGroupingKeys here?

Using new LinkedHashMap<>(customGroupingKeys) would avoid accidental external mutation after construction and also guarantees stable iteration order.

featzhang · 2026-03-17T13:49:28Z

.../src/main/java/org/apache/flink/metrics/prometheus/PrometheusPushGatewayReporterOptions.java

            ConfigOptions.key("randomJobNameSuffix")
                    .booleanType()
-                    .defaultValue(true)
+                    .defaultValue(false)


Changing the default from true to false introduces a behavioral change for existing deployments.

Even though the option is deprecated, preserving the current default for one deprecation cycle may be safer, then switching defaults in a major compatibility window.

featzhang · 2026-03-17T13:50:02Z

.../src/main/java/org/apache/flink/metrics/prometheus/PrometheusPushGatewayReporterOptions.java

+                    .defaultValue(false)
                    .withDescription(
-                            "Specifies whether a random suffix should be appended to the job name.");
+                            "Specifies whether random suffixing `job` label (the old way) to avoid metric collision among reporters from taskamangers."


The option description could use a wording pass:

taskamangers -> taskmanagers

random suffixing job label reads a bit awkwardly

Current wording is understandable, but polishing it would improve config readability.

qinghui-xu force-pushed the FLINK-21309 branch from 3c79d1b to 7a55c41 Compare March 6, 2026 17:29

rionmonster reviewed Mar 8, 2026

View reviewed changes

github-actions bot added the community-reviewed PR has been reviewed by the community. label Mar 9, 2026

Followup: address PR feedback

6fbca7e

- Keep `randomJobNameSuffix` config as deprecated - When `randomJobNameSuffix` is disabled (by default), group metrics using random reporter id

qinghui-xu force-pushed the FLINK-21309 branch from cad1f50 to 6fbca7e Compare March 10, 2026 10:01

featzhang reviewed Mar 17, 2026

View reviewed changes

Conversation

qinghui-xu commented Mar 6, 2026

What is the purpose of the change

Brief change log

Verifying this change

Does this pull request potentially affect one of the following parts:

Documentation

Uh oh!

flinkbot commented Mar 6, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

CI report:

Uh oh!

qinghui-xu commented Mar 7, 2026

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

rionmonster Mar 8, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

rionmonster Mar 8, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

flinkbot commented Mar 6, 2026 •

edited

Loading

rionmonster Mar 8, 2026 •

edited

Loading

rionmonster Mar 8, 2026 •

edited

Loading